Remote device control via transparent display

ABSTRACT

Aspects of the present invention disclose a method for controlling a device remotely. The method includes one or more processors identifying one or more electronic devices, viewed through a transparent display, connected to a network. The method further includes one or more processors determining a first electronic device, from the one or more electronic devices. The method further includes one or more processors mirroring the user interface of the first electronic device on the transparent display.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of remote controldevices, and more particularly to controlling one device using anotherdevice.

Many of today's electronic devices, such as televisions (TV), digitalvideo recorders (DVR), stereo systems, air conditioning (AC) units,ceiling fans, etc., come equipped with remote controls. However, lostremote controls or inaccessible devices are two problems where analternate solution to the standard remote control is desirable. A devicecan be inaccessible for a number of reasons: an individual is bedriddendue to an illness or accident; a child cannot reach a device, like amini-split ductless AC unit, positioned high on a wall; devices arestored in a locked case as a means of loss prevention; etc.

Many other electronic devices, such as smartphones, smartwatches,personal digital assistants (PDA), phablets, tablet computers, etc., donot come equipped with a remote control. Nonetheless, it may beworthwhile to control one of the devices remotely. Consider a retailstore selling electronic devices such as smartphones. Merchandise keptin a secure display case is out-of-reach of potential customers. Thiscreates a problem of the potential customers being unable to directlyinteract with a device under consideration for purchase, which maynegatively impact store profit.

Two modes of device to device communication are wired communication andwireless communication. Regarding wired device to device communicationtechnologies, U.S. Patent 20140375586 A1, titled Function of Touch PanelDetermined by User Gaze, states the following: “A portable mobilecommunication device may be connected to an electronic device using oneor more wire-based technologies. An example of a wire-based technologyis high-definition multimedia interface (HDMI). When the portable mobilecommunication device is connected to an electronic device, the userinterface of the portable mobile communication device may be displayedon the electronic device. A user may interact with the displayed userinterface using the portable mobile communication device.

Concerning wireless communication technologies, the referenced U.S.Patent states: “In some embodiments, a portable mobile communicationdevice may be connected to an electronic device using one or morewireless technologies. Exemplary wireless technologies includenear-field communication (NFC), infra-red transmission (IR), Bluetooth,short-range radio frequency (RF) signals, IEEE 802.11x, WiFi, wirelessuniversal serial bus (USB), HomeRF, IrDA-based (Infrared DataAssociation) transmission, etc. The invention is not limited to anywired or wireless technologies described herein. When a portable mobilecommunication device is connected to an electronic device either via awired or wireless technology, the portable mobile communication devicemay operate in one of two modes.”

SUMMARY

Aspects of the present invention disclose a method for controlling adevice remotely. In one aspect, the method includes one or moreprocessors identifying one or more electronic devices, viewed through atransparent display, connected to a network. The method further includesone or more processors determining a first electronic device, from theone or more electronic devices. The method further includes one or moreprocessors mirroring the user interface of the first electronic deviceon the transparent display.

In another aspect, a computer program product includes programinstructions to identify one or more electronic devices viewed through atransparent display connected to a network. The computer program productalso includes program instructions to determine a first electronicdevice of the one or more electronic devices. The computer programproduct also includes program instructions to mirror the user interfaceof the first electronic device on the transparent display.

In yet another aspect, a computer system includes program instructionsto identify one or more electronic devices viewed through a transparentdisplay connected to a network. The computer system also includesprogram instructions to determine a first electronic device of the oneor more electronic devices. The computer system also includes programinstructions to mirror the user interface of the first electronic deviceon the transparent display.

In yet another aspect, a method of determining a first electronic deviceof the one or more electronic devices includes determining a point ofgaze on the transparent display of a user. The method also includesdetermining a touch point on the transparent display of the user. Themethod also includes determining a straight line using the point of gazeand the touch point. The method also includes determining the firstelectronic device of the one or more electronic devices identified bythe straight line.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of a computer environment depictingan electronic device with a user interface, in accordance with anembodiment of the present invention.

FIG. 2A and FIG. 2B is a flowchart depicting operational steps forselecting a device to replicate on a transparent touchscreen andoperating the device remotely with the transparent touchscreen, inaccordance with an embodiment of the present invention.

FIG. 3 is a block diagram of the components of a computing systemrepresentative of the electronic device of FIG. 1, in accordance with anembodiment of the present invention.

DETAILED DESCRIPTION

Embodiments of the present invention recognize there are instances whencontrolling an electronic device remotely is beneficial. It may benecessary to keep the electronic device secure or the electronic devicemay not be readily accessible due to a physical limitation.

Embodiments of the present invention recognize that there may be a needto remotely control an electronic device, even when the device does nothave a dedicated remote control. Examples of when this may be necessaryare when securing a device from theft or loss, when a device isinaccessible because of its location or a physical limitation, when itis useful to transfer a set of instructions from one device to anotherdevice, or any number of similar situations. One embodiment of thepresent invention allows for selecting an electronic device by simplyviewing it in a display and touching the display's touchscreen. Thisaction will load the software necessary to control the selected deviceto the display. The display is able to control the selected device justas if the device itself was the display. In another embodiment, thedisplay can transfer a set of instructions, for executing a specifictask, from one device to another device. One example of when this typeof instruction transfer may be helpful is transferring an instructionset, seen on a TV show, for washing a particularly delicate fabric, to awashing machine for later use when washing said fabric.

The present invention will now be described in detail with references tothe Figures. FIG. 1 is a functional block diagram of a computerenvironment, generally designated 100, in accordance with an embodimentof the present invention.

An embodiment of computer environment 100 includes network 110, device120, device 130A, and transparent display 140 (TD 140). Computerenvironment 100 can include other computing devices not shown, such asprocessors, hard drives, read only memory, random access memory,displays, tablet computers, laptop computers, desktop computers,computer servers, or any other computer systems known in the art.

In example embodiments, TD 140 may connect to network 110 which enablesTD 140 to access other computing devices and/or data not directly storedto TD 140. Network 110 may be a local area network (LAN), atelecommunications network, a wide area network (WAN) such as theInternet, or any combination of the three, and include wired, wirelessor fiber optic connections. In general, network 110 can be anycombination of connections and protocols that will supportcommunications between TD 140 and other computing devices, in accordancewith embodiments of the present invention.

Device 120 and device 130A are representative electronic devices capableof connecting to network 110. Device 120 and device 130A also includefunctionality that may be mapped to a touchscreen enabled device, suchas TD 140. In one embodiment of the present invention, mapping may bedefined as enabling similar function from one device to another devicewhere the second device does not originally include the mapped function.An example of mapping is enabling the touchscreen functions of asmartphone to a different touchscreen device. Examples of device 120 anddevice 130A include: smartphones; smartwatches; phablets; PDAs; tabletcomputers; laptop computers; TVs; DVRs; various kitchen electronics(e.g., ovens, convection ovens, microwave ovens, bread makers, mixers,blenders, etc.); other home electronics (e.g., washing machines, dryers,thermostats, HVAC (heating, venting and air conditioning) units, ceilingfans, water heaters, etc.); home automation units; and other similaritems. In general, device 120 and device 130A may be any electronicdevice capable of connecting to a network and having touchscreenfunctionality.

Replicating some electronic device functions via mapping to atouchscreen may not be easy to accomplish. One such function difficultto replicate may be the use of accelerometers built into a tabletcomputer. These accelerometers, used in game software to mimic steering,require tilting the tablet left to right and back again. Directlyreplicating this capability is not straight forward. Two possibleoptions for this, and similar situations, are to disable these featureson the touchscreen or to enable them via mapping alternate functions.Consider again the tablet accelerometers; it may be possible to map anoutline of the tablet to the touchscreen and “tilt” the tablet bytouching the left or right side of the replicate to steer left or right.Another example of a function that cannot be mapped is one that requiresadditional hardware not found in the touchscreen such as attempting touse the touchscreen as a GPS (Global Positioning System) unit. Assume,in this example, that TD 140 does not include a GPS chip. Because TD 140does not include the GPS chip, while the tablet in the above exampledoes, the GPS function cannot be mapped to the touchscreen.

In various embodiments of the present invention, TD 140 is a touchscreenenabled transparent display with the capability to map the touch enabledfunctions of another electronic device viewed through the transparentdisplay. A touchscreen is an electronic visual display that may allow auser to control a device through simple or multi-touch gestures bytouching the screen with a special stylus/pen and/or one or morefingers/fingernails. A touchscreen allows a user to interact directlywith what is displayed, rather than using a mouse, touchpad or any otherintermediate device (with the exception of a stylus which certain typesof touchscreens require). The user may use the touchscreen to react towhat is displayed and to control how it is displayed (i.e., increasingthe text size to improve readability). In one embodiment, TD 140 may beessentially a transparent piece of glass. In another embodiment, TD 140may be a transparent piece of glass held by a frame; the frame may bemetal, plastic, composite, or another material. In yet anotherembodiment, TD 140 may be a display case such as those used in a retailenvironment. An example of this embodiment may be a display case in astore that markets cell phones. TD 140 may include components asdepicted and described in further detail with respect to FIG. 3, inaccordance with embodiments of the present invention.

TD 140 includes replicate device 130B, user interface 150, datarepository 160, front-side image device 170, back-side image device 175,gaze program 180, touch sensor 190, and display program 195. Thetransparent nature of TD 140 enables a user to look through TD 140 as ifthe user were looking through glass. In one embodiment of the presentinvention, TD 140 allows a user to select an electronic device viewedthrough TD 140 and displayed in user interface 150 by using the featuresof front-side image device 170, back-side image device 175, gaze program180, touch sensor 190, and display program 195. In another embodiment,TD 140 may store instructions from a first selected electronic device todata repository 160 and transmit said instructions to a second selectedelectronic device. In yet another embodiment, TD 140 can execute thereplicated touch functions, of a selected electronic device viewedthrough the display, on user interface 150 using display program 195.

Device 130B is a replicate of device 130A displayed on TD 140, inaccordance with embodiments of the present invention. Device 130Breplicates all of the functionality of device 130A that is able to bemapped to a touchscreen device such as TD 140. There may be somelimitations in the mapped functionality based on the technology found indevice 130A. For example, consider viewing a replicate tablet in TD 140.Many touch functions of the original tablet may be mapped to TD 140.Some examples of touch functions that may be mapped include: tapping anicon to open a software application (or ‘app’), typing on a virtualkeyboard, selecting and playing audio files, using a text editor orspreadsheet program, using a web browser, etc.

In example embodiments, user interface 150, used in conjunction with TD140, provides an interface between a user of TD 140, network 110 and anyother devices connected to network 110. In general, a user interface isthe space where interactions between humans and machines occur. Userinterface 150 may be a graphical user interface (GUI) or a web userinterface (WUI) and can display text, documents, web browser windows,user options, application interfaces, and instructions for operation,and include the information (such as graphic, text, and sound) that aprogram presents to a user and the control sequences the user employs tocontrol the program. User interface 150 may also be mobile applicationsoftware that provides an interface between a user of TD 140 and network110. Mobile application software, or an “app,” is a computer programdesigned to run on smartphones, phablets, tablet computers and othermobile devices. User interface 150 enables a user of TD 140 to receivean indicator of one or more previous viewing locations and a summary ofviewing history. In various embodiments of the present invention, it ispossible to divide user interface 150 into different and distinctsections. As illustrated in FIG. 1, dividing user interface 150 in halfcreates two areas in the display. In other embodiments, user interface150 may display 4 areas (quadrants), 6 areas, or any number of displayareas (limited in practicality by the physical size of TD 140 and theresolution of user interface 150). User interface 150 may include anoptical coating, an oleophobic coating, a screen protector, or the like,all of which help to mitigate the presence of fingerprints on userinterface 150.

In various embodiments, data repository 160 may be storage that may bewritten to or read by display program 195 for use by display program195. Data repository 160 may be implemented with various types ofstorage such as random access memory (RAM), a hard disk drive (HDD), asolid state drive (SSD, flash memory, etc. In one embodiment, datarepository 160 resides on TD 140. In alternate embodiments, datarepository 160 may also be cloud storage accessed over network 110 ordata repository 160 may reside on another computing device withincomputer environment 100. Device operating instructions, for example,those allowing touch screen enabled functionality, may be stored to datarepository 160. For example, instructions for launching a calendar appfor a smartphone may be stored to data repository 160.

In example embodiments, front-side image device 170 enables TD 140 todetermine the point of gaze of a user of TD 140; i.e., where the user islooking on user interface 150 of TD 140. In one embodiment, front-sideimage device 170 may be part of a microprojector which transmits aninfrared (IR) beam at the eye, and a set of sensors receive thereflection patterns to determine eye position and movement. In anotherembodiment, front-side image device 170 may be a video camera whichreceives the reflection patterns from the eye. Video based eye trackersmay track corneal reflection and the center of the eye's pupil, maytrack reflections from the front of the eye's cornea and the back of theeye's lens, or may track features inside the eye, such as retinal bloodvessels, as the eye moves. In general, front-side image device 170 isany device capable of tracking eye movement.

In various embodiments of the present invention, back-side image device175 displays an image of the surrounding environment in user interface150 on TD 140. Back-side image device 175 shows what electronic devicesare in the environment as well as their relative positions to oneanother. The relative positions of the various electronic devices in theenvironment provides another form of validation for display program 195in selecting which electronic device will be controlled by TD 140. Oncerelative position is determined and stored to data repository 160, theunique identifiers (e.g., IP address, MAC address, etc.) stored in adatabase (not shown) connected to network 110 are known for eachelectronic device in the environment. In one embodiment, back-side imagedevice 175 is a camera capable of capturing still images while inanother embodiment, back-side image device 175 is a video camera capableof recording video files. Back-side image device 175 allows a user of TD140 to touch an image shown on TD 140 as a means of selecting anelectronic device for remote control. In various embodiments, TD 140 mayinclude one, or more than one, devices such as back-side image device175. In general, back-side image device 175 is any device capable ofdisplaying an image.

Gaze program 180 is a software application that determines the point ofgaze of a user of TD 140 (i.e., where the user is looking on userinterface 150 of TD 140) based on the input received by front-side imagedevice 170. In various embodiments, gaze program 180 is included in eyetracker systems, such as front-side image device 170. In otherembodiments, gaze program 180 is a stand-alone software program whichworks in conjunction with front-side image device 170. In general, gazeprogram 180 is any set of program instructions capable of determiningthe point of a user's gaze with input from an image device such asfront-side image device 170.

According to various embodiments of the present invention, touch sensor190 is a technology used in conjunction with user interface 150,allowing TD 140 to accept user input. The often used name, known bythose skilled in the art, for touch sensor 190 is a touchscreen.Touchscreens are included in many electronic devices such as gameconsoles, tablet computers, phablets, smartphones, PDAs, and the like.In example embodiments, touch sensor 190 may include the followingtechnologies: resistive; surface acoustic wave; capacitive, includingsurface capacitance; projected capacitance; mutual capacitance; andself-capacitance; infrared grid; infrared acrylic projection; opticalimaging; dispersive signal technology; acoustic pulse recognition; andthe like. In one embodiment, touch sensor 190 may include hapticfeedback, an example of which is a vibratory response to the touch ofuser interface 150.

In example embodiments, display program 195 is an application capable ofoperating a replicated device displayed on user interface 150 of TD 140.Display program 195 may access data repository 160 to locate the devicespecific instructions to operate the replicated device. Display program195 may also access a database (not shown) to locate specific electronicdevice identifiers or operating instructions. In one embodiment, displayprogram 195 may store a file (or files) from device 130A to datarepository 160 and subsequently, transfer the file (or files) to device120 to execute on that device. This process is analogous to a cut andpaste operation in a word processing application (copying informationfrom one location and transferring the information to another location).In an example, the file can be an execution file, which is a set ofinstructions written to perform a specific task; an example discussedlater is an energy saving program for a user programmable thermostat.

FIG. 2A and FIG. 2B are a flowchart depicting operational steps 200 ofdisplay program 195, an application which allows a user of TD 140 toselect a device viewed on user interface 150 and to operate functions ofthe device via user interface 150. In one embodiment display program 195initiates when TD 140 is powered on. In another embodiment, displayprogram 195 initiates when TD 140 detects devices connected to network110. In yet another embodiment, a user, via user interface 150, mayinitiate display program 195 by touching the icon associated withdisplay program 195 on user interface 150.

Display program 195 detects and connects to a network (step 202). Inother words, display program 195 may detect what networks are availableand connect to them. In one embodiment, display program 195 detects asignal broadcast by network 110, which may be a home based networkcreated by a Wi-Fi router. In another embodiment, display program 195receives input from touch sensor 190 indicating that a wireless networkicon, displayed on user interface 150, has been selected by a user'stouch. In response, display program 195 lists available networks.Display program 195 then receives input, from touch sensor 190, of theuser selecting one of the available networks by touching that network'sicon displayed in user interface 150. Display program 195 then receivesinput, from touch sensor 190, that the user has entered a security keyvia a virtual keyboard displayed in user interface 150. Display program195 then receives input, from touch sensor 190, that the user hastouched the area of user interface 150 corresponding to the connect iconfor the network. Successfully completing this process connects TD 140 tothe selected network.

Display program 195 focuses on the surrounding environment (step 204).In other words, display program 195 receives input from a user of TD 140to focus back-side image device 175 on the surrounding environment inpreparation for selecting an electronic device to be replicated in userinterface 150 of TD 140. In one embodiment, back-side image device 175sends input to display program 195 regarding the environment back-sideimage device 175 is viewing. In another embodiment, back-side imagedevice 175 is one of a plurality of image devices on a display case in aretail environment. In an example, a user focuses TD 140 on a smartphoneand a tablet computer sitting on the coffee table in the user's livingroom.

Display program 195 identifies devices on the network (step 206). Inother words, display program 195 identifies electronic devices connectedto the network and attempts to establish a link between the electronicdevices and TD 140. In an embodiment of the present invention, displayprogram 195 receives input identifying electronic devices connected tonetwork 110. In one embodiment, if no security key is required, displayprogram 195 may immediately establish a link to an electronic device. Inanother embodiment, display program 195 receives input from anelectronic device that a security key is required. Display program 195then receives the necessary credentials (e.g., IP address, MAC address,hostname, User ID, password, etc.), via touch sensor 190 registeringtouches on a virtual keyboard displayed on user interface 150, toestablish the link. The IP address for a given electronic device may befound in the details of the electronic device's network connection andthe MAC address may be found in the details of the electronic device'slocal area connection. In an example, a user of TD 140 defines asmartphone by its IP address and enters a user ID and password, asneeded.

Display program 195 receives input from gaze program 180 (step 208). Inother words, gaze program 180 determines the point of gaze on TD 140 bya user and sends that input to display program 195. In an exampleembodiment, a video camera, such as front-side image device 170, tracksthe user's eye movements to determine where the user is looking on TD140. In one example, the user is looking at a smartphone through TD 140.The smartphone, connected to the same Wi-Fi network as TD 140, isdisplaying a program for a programmable thermostat.

Display program 195 receives input from touch sensor 190 (step 210). Inother words, touch sensor 190 determines the x-y coordinates of a user'stouch contact point on user interface 150 and sends that input todisplay program 195. In an embodiment of the present invention, acapacitive touchscreen detects a user's touch because of the conductivenature of the human body. The user's touch distorts the electrostaticfield of the touchscreen measured by a change in capacitance. Touchsensor 190 determines the location of the touch by the user and sendsthe information to display program 195. In an example, a user of TD 140touches the point on user interface 150 that corresponds to the locationthat the user is looking through TD 140 to see a smartphone behind TD140.

Display program 195 extrapolates a straight line (step 212). In otherwords, display program 195 creates a straight line between the twopoints created by the point of gaze determined in step 208 and the touchcontact point determined in step 210. From this extrapolation, displayprogram 195 determines the electronic device a user of TD 140 isviewing. Display program 195 recognizes the electronic device by usingone of any number of object recognition techniques available. Objectrecognition, or object detection, is a computer technology concerningcomputer vision and image processing related to detecting instances ofobjects of a certain class. In one embodiment, display program 195receives input from gaze program 180 and touch sensor 190 establishingthe two points to create the extrapolation. In an example, displayprogram 195 determines a user is viewing a smartphone through TD 140 byextrapolating the line created by the two points and determining thatthe line intersects the smartphone.

Display program 195 mirrors the selected electronic device's userinterface (step 214). In other words, the selected electronic devicedisplays its user interface and display program 195 mirrors a copy ofthis user interface on user interface 150 of TD 140. In one embodiment,the mirrored user interface may be an exact copy while in alternativeembodiments, the mirrored user interface may be substantially similar.This enables a user of TD 140 to control the replicated device usingonly TD 140 without physically handling the replicated device itself.Mirroring, which may be defined as displaying the content of a firstdevice (the selected electronic device) on a second device (TD 140),requires the use of the first device's program software by the seconddevice. In one embodiment, the program software is preloaded to datarepository 160 for use by TD 140 while in another embodiment, theselected electronic device's program software is loaded to userinterface 150 directly from the replicated device. In yet anotherembodiment, the program software is loaded from a database (not shown)connected to network 110. In various embodiments of the presentinvention, TD 140 communicates with the selected electronic device usingWi-Fi, Bluetooth, Near Field Communication (NFC), or any other availablewireless communication technology. In an example, the program softwarefor a smartphone, viewed through TD 140, is loaded to TD 140 allowingremote control of the smartphone using the smartphone's mirrored userinterface on user interface 150 of TD 140.

Display program 195 operates the replicate device (step 216). In otherwords, a user of TD 140 controls the replicated device via displayprogram 195; the selected electronic device is replicated on TD 140 anddisplayed on user interface 150. In one embodiment, display program 195receives input, via user interface 150, to open an application on thereplicated device. In an example, a user of TD 140 is able to open asmartphone's contact list or launch its calendar app through userinterface 150 on TD 140. In another example, the user may launch thebrowser on the smartphone, use a search function to find a videotutorial for using a programmable thermostat and watch the video, all bycontrolling the actual smartphone through the replicated smartphonedisplayed in user interface 150 on TD 140.

Display program 195 determines whether to transfer a file or files(decision step 218). In other words, display program 195 receives inputfrom a user of TD 140, via user interface 150, whether to transfer anyfiles that have been previously stored to data repository 160 or anyother storage device. The files may have been copied earlier from otherreplicated devices for later transfer to another replicated device. Inone embodiment, (decision step 218, NO branch), the available file willnot be copied for subsequent transfer; therefore, display program 195proceeds directly to step 224. In another embodiment, (decision step218, YES branch), a file will be copied for subsequent transfer; displayprogram 195 then proceeds to step 220.

Display program 195 determines whether to connect to a new device(decision step 220). In other words, display program 195 receives inputfrom a user of TD 140, via user interface 150, whether to connect to anew electronic device. In one embodiment, the new electronic device isconnected to network 110. In another embodiment, the new electronicdevice is on a different network. In one embodiment (decision step 220,NO branch), display program 195 does not receive an indication from auser of TD 140 to connect to a new electronic device; therefore, displayprogram 195 proceeds to step 222. In another embodiment (decision step220, YES branch), display program 195 receives input from a user of TD140 to connect to a new electronic device, display program 195 thenproceeds to step 202. In one embodiment, the currently selected devicereplicated in TD 140 will remain active (i.e., will remain replicated inTD 140 which allows control of the replicated device via user interface150). This will result in multiple replicated devices on TD 140 (afterthe new electronic device is replicated). In another embodiment, thecurrently selected device replicated in TD 140 will be disconnectedresulting in TD 140 losing control capability of that replicated device.

Display program 195 transfers a file or files (step 222). In otherwords, display program 195 receives input from a user of TD 140, viauser interface 150, that a file or files stored in data repository 160or stored in another storage device will be transferred to anotherelectronic device. In one embodiment, the file or files are transferredto the currently replicated device on TD 140. The files, therefore, werepreviously copied from a different electronic device that, at one time,was replicated on TD 140. In another embodiment, the files aretransferred to an electronic device not currently replicated on TD 140.This first requires that a connection is made to a new electronic device(decision step 220, YES branch). In yet another embodiment, multipleelectronic devices are currently replicated on TD 140 so the file orfiles can be transferred directly between the electronic devices.

Display program 195 determines whether to copy a file (decision step224). In other words, display program 195 receives input that a selectedelectronic device, controlled by TD 140, includes a file. In oneembodiment, (decision step 224, NO branch), the available file will notbe copied; therefore, display program 195 proceeds directly to step 228.In another embodiment, (decision step 224, YES branch), a file will becopied; display program 195 then proceeds to step 226.

Display program 195 stores a file (step 226). In other words, displayprogram 195 receives input from a user of TD 140 to store a file. In oneembodiment, display program 195 receives input from a user of TD 140 tostore a file (or files) to data repository 160 for later use or latertransfer to another electronic device. In various embodiment, the fileor files to be stored may be stored to any storage device connected tonetwork 110. In an example, a user of TD 140 finds an execution file,outlining a program, in a video tutorial for using a programmablethermostat, and stores the execution file to data repository 160 tolater transfer to a programmable thermostat. In another example, a userof TD 140 stores music files to data repository 160. This exampleillustrates that not all files stored on TD 140 must later betransferred to another electronic device but rather may be used directlyby TD 140.

Display program 195 operates the replicate device(s) (step 228). Inother words, a user of TD 140 controls the selected electronic device(s)via display program 195; the selected electronic device is (or devicesare) replicated on TD 140 and displayed on user interface 150. In oneembodiment, display program 195 receives input to load a smartphone'sprogram software to user interface 150. In an example, a user of TD 140is able to open a smartphone's contact list or launch its calendar appthrough user interface 150 on TD 140. In another example, the user maylaunch the browser on the smartphone, use a search function to find avideo tutorial for using a programmable thermostat and watch the video,all by controlling the actual smartphone through the replicatedsmartphone displayed in user interface 150 on TD 140.

Display program 195 determines whether to select another device(decision step 230). In other words, display program 195 receives input,from a user of TD 140, regarding the selection of another device forcontrol by TD 140. In one embodiment, (decision step 230, YES branch),display program 195 receives input to select another device for remotecontrol by TD 140 so display program 195 proceeds to step 202. In anexample, a user of TD 140 intends to transfer an execution file for aprogrammable thermostat so the user locates the programmable thermostaton the network, gains control of a replicated version of theprogrammable thermostat using steps 202-216, inclusive. The user is thenable to transfer the execution file, stored in data repository 160, tothe programmable thermostat using only TD 140 and not physicallytouching the programmable thermostat. In another embodiment, (decisionstep 230, NO branch), display program 195 does not receive input toselect another device for remote control so display program 195 proceedsto step 232. In an embodiment of the present invention, the currentlyselected device replicated in TD 140 will remain active (i.e., willremain replicated in TD 140 which allows control of the replicateddevice via user interface 150). This will result in multiple replicateddevices on TD 140 (after the new electronic device is replicated). Inanother embodiment, the currently selected device replicated in TD 140will be disconnected resulting in TD 140 losing control capability ofthat replicated device. In an example, the user of TD 140 continues towork with a smartphone's calendar app via the replicate displayed inuser interface 150 on TD 140 while in another example, the smartphonereplicate is disconnected so remote control of the smartphone is lost.

Display program 195 closes the program (step 232). In other words,display program 195 continues to operate any selected electronic devicesvia TD 140 until a user of TD 140 closes display program 195. In oneembodiment, display program 195 continues to control the replicatedsmartphone via TD 140. In an example, a user of TD 140 is able to open asmartphone's contact list or launch its calendar app through userinterface 150 on TD 140. In another example, the user may launch abrowser on the smartphone, use a search function to find a videotutorial for using a programmable thermostat and watch the video, all bycontrolling the actual smartphone through the replicated smartphonedisplayed in user interface 150 on TD 140. In another embodiment, a userof TD 140 closes display program 195. In an example, control of anyreplicated devices linked to TD 140 is lost.

FIG. 3 depicts computer system 300, that is an example of a system thatincludes TD 140. Computer system 300 includes processors 304, cache 314,memory 306, persistent storage 308, communications unit 310,input/output (I/O) interface(s) 312 and communications fabric 302.Communications fabric 302 provides communications between cache 314,memory 306, persistent storage 308, communications unit 310, andinput/output (I/O) interface(s) 312. Communications fabric 302 can beimplemented with any architecture designed for passing data and/orcontrol information between processors (such as microprocessors,communications and network processors, etc.), system memory, peripheraldevices, and any other hardware components within a system. For example,communications fabric 302 can be implemented with one or more buses or acrossbar switch.

Memory 306 and persistent storage 308 are computer readable storagemedia. In this embodiment, memory 306 includes random access memory(RAM). In general, memory 306 can include any suitable volatile ornon-volatile computer readable storage media. Cache 314 is a fast memorythat enhances the performance of processors 304 by holding recentlyaccessed data, and data near recently accessed data, from memory 306.

Program instructions and data used to practice embodiments of thepresent invention may be stored in persistent storage 308 and in memory306 for execution by one or more of the respective processors 304 viacache 314. In an embodiment, persistent storage 308 includes a magnetichard disk drive. Alternatively, or in addition to a magnetic hard diskdrive, persistent storage 308 can include a solid state hard drive, asemiconductor storage device, read-only memory (ROM), erasableprogrammable read-only memory (EPROM), flash memory, or any othercomputer readable storage media that is capable of storing programinstructions or digital information.

The media used by persistent storage 308 may also be removable. Forexample, a removable hard drive may be used for persistent storage 308.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer readable storage medium that is also part of persistent storage308.

Communications unit 310, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 310 includes one or more network interface cards.Communications unit 310 may provide communications through the use ofeither or both physical and wireless communications links. Programinstructions and data used to practice embodiments of the presentinvention may be downloaded to persistent storage 308 throughcommunications unit 310.

I/O interface(s) 312 allows for input and output of data with otherdevices that may be connected to each computer system. For example, I/Ointerface 312 may provide a connection to external devices 316 such as akeyboard, keypad, a touch screen, and/or some other suitable inputdevice. External devices 316 can also include portable computer readablestorage media such as, for example, thumb drives, portable optical ormagnetic disks, and memory cards. Software and data used to practiceembodiments of the present invention can be stored on such portablecomputer readable storage media and can be loaded onto persistentstorage 308 via I/O interface(s) 312. I/O interface(s) 312 also connectto display 318.

Display 318 provides a mechanism to display data to a user and may be,for example, a computer monitor.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium can be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The terminology used herein was chosen to best explain the principles ofthe embodiment, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments disclosed herein.

What is claimed is:
 1. A method for controlling a device remotely, themethod comprising: identifying, by one or more processors, a pluralityof electronic devices, viewed through a transparent display having touchfunctionality, connected to a network, wherein the transparent displayallows a user to view each electronic device of the plurality ofelectronic devices through the transparent display: selecting, by one ormore processors, a first electronic device of the plurality ofelectronic devices by: receiving a plurality of inputs from the user onthe transparent display; determining a point of gaze of the user on thetransparent display based on the received plurality of inputs;determining a touch point on the transparent display based on thereceived plurality of inputs; extrapolating a straight line between thedetermined point of gaze and the touch point determined on thetransparent display; and selecting the first electronic deviceintersected by the extrapolated straight line through the transparentdisplay: identifying, by one or more processors, the selected firstelectronic device via object recognition; mirroring, by one or moreprocessors, a user interface of the identified first electronic deviceand displaying content of the first electronic device on the mirroreduser interface of the first electronic device displayed on thetransparent display using program software of the first electronicdevice, wherein the mirrored user interface is a copy of the userinterface of the first electronic device that copies all functionalityof the first electronic device and maps to the transparent display,wherein program software of the first electronic device, viewed throughthe transparent display, is loaded to the transparent display from thecopy of the user interface of the first electronic device, or from adatabase connected to the network, and the mirrored user interface ofthe first electronic device displayed on the transparent display allowsthe user to remotely control the first electronic device viewed throughthe transparent display.
 2. The method of claim 1, further comprising:storing, by one or more processors, a set of data from the firstelectronic device on the transparent display.
 3. The method of claim 2,further comprising: selecting, by one or more processors, a secondelectronic device of the plurality of electronic devices; mirroring, byone or more processors, a user interface of the second electronic deviceon the transparent display; and transferring, by one or more processors,the set of data to the second electronic device.
 4. The method of claim1, wherein the step of determining a point of gaze of the user on thetransparent display based on the received plurality of inputs,comprises: receiving, by one or more processors, a set of input datafrom an image device; and processing, by one or more processors, the setof input data to determine the point of gaze on the transparent display.5. The method of claim 1, wherein the step of determining the touchpoint on the transparent display based on the received plurality ofinputs, comprises: receiving a set of input data from a touch sensor;and processing the set of input data to determine the touch point on thetransparent display.
 6. A computer program product for controlling adevice remotely, the computer program product comprising: one or morecomputer readable hardware storage media; and program instructionsstored on the one or more computer readable hardware storage media forexecution by at least one or more computer processors, the programinstructions comprising: program instructions to identify a plurality ofelectronic devices, viewed through a transparent display having touchfunctionality, connected to a network, wherein the transparent displayallows a user to view each electronic device of the plurality ofelectronic devices through the transparent display; program instructionsto select a first electronic device of the plurality of electronicdevices, wherein the program instructions to select a first electronicdevice of the plurality of electronic devices comprise: programinstructions to receive a plurality of inputs from the user on thetransparent display; program instructions to determine a point of gazeof the user on the transparent display based on the received pluralityof inputs; program instructions to determine a touch point on thetransparent display based on the received plurality of inputs; programinstructions to extrapolate a straight line between the point of gazeand the touch point determined on the transparent display; and programinstructions to select the first electronic device intersected by theextrapolated straight line through the transparent display; programinstructions to identify the selected first electronic device via objectrecognition; program instructions to mirror a user interface of theidentified first electronic device and displaying content of the firstelectronic device on the mirrored user interface of the first electronicdevice displayed on the transparent display using program software ofthe first electronic device, wherein the mirrored user interface is acopy of the user interface of the first electronic device that copiesall functionality of the first electronic device and maps to thetransparent display, wherein program software of the first electronicdevice, viewed through the transparent display, is loaded to thetransparent display from the copy of the user interface of the firstelectronic device, or from a database connected to the network, and themirrored user interface of the first electronic device displayed on thetransparent display allows the user to remotely control the firstelectronic device viewed through the transparent display.
 7. Thecomputer program product of claim 6, further comprising programinstructions, stored on the one or more computer readable hardwarestorage media, to: store a set of data from the first electronic deviceon the transparent display.
 8. The computer program product of claim 7,further comprising program instructions, stored on the one or morecomputer readable hardware storage media, to: select a second electronicdevice of the plurality of electronic devices; mirror a user interfaceof the second electronic device on the transparent display; and transferthe set of data to the second electronic device.
 9. The computer programproduct of claim 6, wherein the program instructions to determine apoint of gaze of the user on the transparent display based on thereceived plurality of inputs, comprises: program instructions to receivea set of input data from an image device; and program instructions toprocess the set of input data to determine the point of gaze on thetransparent display.
 10. The computer program product of claim 6,wherein the program instructions to determine the touch point on thetransparent display, based on the received plurality of inputs,comprise: program instructions to receive a set of input data from atouch sensor; and program instructions to process the set of input datato determine the touch point on the transparent display.
 11. A computersystem for controlling a device remotely, the computer systemcomprising: one or more computer processors; one or more computerreadable hardware storage media; and program instructions stored on theone or more computer readable hardware storage media for execution by atleast one of the one or more computer processors, the programinstructions comprising: program instructions to identify a plurality ofelectronic devices, viewed through a transparent display having touchfunctionality, connected to a network, wherein the transparent displayallows a user to view each electronic device of the plurality ofelectronic devices through the transparent display; program instructionsto select a first electronic device of the plurality of electronicdevices, wherein the program instructions to select a first electronicdevice of the plurality of electronic devices comprise: programinstructions to receive a plurality of inputs from the user on thetransparent display; program instructions to determine a point of gazeof the user on the transparent display based on the received pluralityof inputs; program instructions to determine a touch point on thetransparent display based on the received plurality of inputs; programinstructions to extrapolate a straight line between the point of gazeand the touch point determined on the transparent display; and programinstructions to select the first electronic device intersected by theextrapolated straight line through the transparent display; programinstructions to identify the selected first electronic device via objectrecognition; program instructions to mirror a user interface of theidentified first electronic device and display content of the firstelectronic device on the mirrored user interface of the first electronicdevice displayed on the transparent display using program software ofthe first electronic device, wherein the mirrored user interface is acopy of the user interface of the first electronic device that copiesall functionality of the first electronic device and maps to thetransparent display, wherein program software of the first electronicdevice, viewed through the transparent display, is loaded to thetransparent display from the copy of the user interface of the firstelectronic device, or from a database connected to the network, and themirrored user interface of the first electronic device displayed on thetransparent display allows the user to remotely control the firstelectronic device viewed through the transparent display.
 12. Thecomputer system of claim 11, further comprising program instructions,stored on the one or more computer readable hardware storage media forexecution by at least one of the one or more computer processors, to:store a set of data from the first electronic device on the transparentdisplay.
 13. The computer system of claim 12, further comprising programinstructions, stored on the one or more computer readable hardwarestorage media for execution by at least one of the one or more computerprocessors, to: select a second electronic device of the plurality ofelectronic devices; mirror a user interface of the second electronicdevice on the transparent display; and transfer the set of data to thesecond electronic device.
 14. The computer system of claim 11, whereinthe program instructions to determine a point of gaze of the user on thetransparent display based on the received plurality of inputs, comprise:program instructions to receive a set of input data from an imagedevice; and program instructions to process the set of input data todetermine the point of gaze on the transparent display.
 15. The computersystem of claim 11, wherein the program instructions to determine atouch point on the transparent display based on the received pluralityof inputs, comprise: program instructions to receive a set of input datafrom a touch sensor; and program instructions to process the set ofinput data to determine the touch point on the transparent display.